Repository navigation
docs: when Browser checks and Playwright Check Suites keep traces and videos - #813
Conversation
…aces and videos Add a troubleshooting entry for missing traces and videos in Browser checks, correct the Playwright Check Suite trace default (on-first-retry, retention follows use.trace), and say Browser checks record one video per page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
…eout message Tested on a production account: a passing Browser check run with checkly test --record and checkly trigger --record keeps video and trace. A run past 240 s fails with 'Timed out waiting 240s for the test suite to run'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hlenke
left a comment
There was a problem hiding this comment.
Needs changes · The trace and video claims check out against the runner source. Two things to fix first: the Playwright Check Suite construct page still says traces are kept on every run, and the renamed timeout heading drops an error message some readers still see.
Sibling page that now disagrees (not in the diff). constructs/playwright-check.mdx:157 lists the defaults as workers: 4, retries: 2, use.video: "on-first-retry" and use.screenshot: "on-first-retry", then says "Checkly also records traces on every run". After this PR, detect/synthetic-monitoring/playwright-checks/configuration.mdx lists use.trace: on-first-retry as an overridable default and says the trace is kept only when use.trace would have produced one. A reader of the construct page will expect a trace for every run. Suggested replacement for that line:
Checkly applies these Playwright defaults when you do not set them yourself:
workers: 4,retries: 2,use.trace: "on-first-retry",use.video: "on-first-retry", anduse.screenshot: "on-first-retry". Checkly records a trace on every run and keeps it according to youruse.tracesetting. It also uses the Checkly reporter and appends a Checkly user-agent suffix. Checkly supports Playwright'sglobalTimeout; the timer starts whennpx playwright testbegins.
The two guides that already describe this (guides/monitoring-ecommerce-apps-using-playwright.mdx:179, guides/playwright-testing-to-monitoring.mdx:120) agree with this PR.
Overlap. #818 also edits detect/synthetic-monitoring/browser-checks/troubleshooting.mdx. The hunks don't overlap: this PR adds an accordion after Performance and Timeout Issues; #818 edits the Timing and Race Conditions list and appends a new section at the end. Either order merges cleanly. I'd land this one first because it's smaller and #818's new intro links to the same trace docs. Once both land, the page says twice that pages from browser.newContext() have no video (item 3 here, and #818's "Tests share a page or context from beforeAll"). That's flagged on #818.
CI: all checks pass. Mergeable: clean. main has not changed the touched files since the merge base.
Inline: 1 important.
What I checked
- Read the full diff from the merge base
e0e0277fand the touched pages at the PR head:browser-checks/troubleshooting.mdx,timeouts.mdx,playwright-support.mdx, andplaywright-checks/configuration.mdx. - Local app unavailable (BUSY: another Tilt on the ports), so I verified against source instead of end-to-end runs. The PR description reports real Playwright runs and production test-account runs. I didn't repeat those.
- Browser check trace/video modes: monorepo
mainpackages/check-runtimes-common/src/browser/getPlaywrightTestConfig.jsandpackages/playwright-check-runner/src/browser/browser-runner.ts. Both settraceandvideotoonfor ad-hoc runs andretain-on-failureotherwise, after the user'suse, so the user'strace/videoare overridden. Ad-hoc includes runs with a test session and test result, which is what--recordcreates. This matchesplaywright-support.mdx(trace/videomarked ❌ in the Use tab). - Playwright Check Suite defaults:
packages/playwright-check-runner/src/lib/wrapperHelpers.ts. The defaultuse.traceison-first-retry; recording is forced toon; the requested mode is passed to the reporter as retention. Video and screenshot defaults areon-first-retry, andretriesdefaults to2. These match the new table row and paragraph. - Run-limit message: for
@playwright/testscripts the runner passes the execution timeout as Playwright's global timeout, which yieldsTimed out waiting 240s for the test suite to run. Scripts that don't use@playwright/teststill fail withYour check run has reached the maximum run time of … ms.The run timeout has no Playwright global timeout in front of it there (browser-runner.ts, run-timeout handling; legacyapps/runners/functions/src/browser-check-common/src/AbstractBrowserRuntime.js). - Playwright videos guide (https://playwright.dev/docs/videos): manually created contexts are handled separately from the test fixtures'
videooption. - Siblings: grepped all
.mdxfor trace retention claims (records traces on every run,use.trace,traces are only). scripts/pr_impact.py: no findings. Anchors#viewing-trace-filesand#checkly-applied-playwright-settingsexist at the PR head.- Not verified: the video-per-page and
browser.newContext()claims beyond the Playwright docs; I relied on the author's runs.
What changed and why
People often ask why a Browser check result has no trace or video, how to get them for passing runs, and how to turn video off. The facts were spread across the Playwright support page, and the Playwright Check Suite page described an old trace default.
traceandvideoPlaywright options are ignored (use a Playwright Check Suite to control them), that pages from a manually createdbrowser.newContext()have no video, and that the test timeout should stay below the 240-second run limit.use.tracemoves from "enforced:onfor every run" to the defaults table ason-first-retry. A short paragraph explains that Checkly always records a trace but keeps the file only when youruse.tracesetting would have produced one, and what the defaults mean for passing and failing tests.Timed out waiting 240s for the test suite to run, instead of "maximum run time of 240000 ms". "No assets will be available" becomes: the test still running at the limit has no trace or video.How each claim was verified
Browser check and Playwright Check Suite runners can't run in an isolated local devenv. Each claim was tested with real Playwright 1.58.2 runs, using the config that Checkly's own code generates on monorepo
main(e086a77755):getPlaywrightTestConfigwith the runner's CLI flags (--global-timeout=240000, a kill at 250 s).applyChecklyDefaults.retain-on-failure. Passing run: no artifacts. Failing run:video.webmandtrace.zipon. Passing run:video.webmandtrace.zipuse.trace/use.videouse: { video: 'off', trace: 'off' }into the generator still yieldsretain-on-failure/on(same in the runner-ng copy,browser-runner.ts)context.newPage():video.webmandvideo-1.webmbrowser.newContext()pages have no videotrace.zipfiles and reported videos (browser-assets.ts)use.traceison-first-retry; recording forced toonapplyChecklyDefaultsoutput (trace: 'on', retentionon-first-retry). Unit tests:wrapperHelpers.test.ts(114 pass)use.tracetrace-extractiontests (37 pass,shouldKeepTrace).use: { trace: 'off' }gives retentionoff,trace: 'on'giveson--retries=0: no video, trace droppeduse.video: 'off'turns video offoffnpx checkly test --recordandnpx checkly trigger --recordkeep the trace and video of a passing Browser checkGET /v1/test-sessions/{id}/results/{resultId}/assetslistsvideo.webmand the trace zip for bothTimed out waiting 240s for the test suite to run; the test still running has no trace or videoTimed out waiting 240s for the test suite to runand… for the teardown for test suite to run; assets hold only the first test's video and traceassets:read(already in the permissions reference, not changed here); every built-in role has itassets:read(withchecks:read) gets 403 from the web app's asset routes (screenshot signed URL, navigation trace read) and from the public API (/next/assets/…screenshot link and trace read,/v1/check-results/{checkId}/{checkResultId}/assets); the owner gets 200 from the asset reads. The same member moved to Read Only, Read & Run, Read & Write and Admin gets 200 from each asset read (the/v1/check-resultsmanifest passes the permission check and answers 404, as the seeded asset has no stored result). Built-in role lists on monorepomain: all five roles holdassets:readNot tested / left out
--recordand run-limit claims were tested with real runs on a production test account.assets:read(View result assets) since docs: add a complete permissions reference #581; the test is in the table above.🤖 Generated with Claude Code